{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Datawhale 零基础入门数据挖掘-Task5 模型融合\n",
    "\n",
    "## 五、模型融合\n",
    "\n",
    "Tip:此部分为零基础入门数据挖掘的 Task5 模型融合 部分，带你来了解各种模型结果的融合方式，在比赛的攻坚时刻冲刺Top，欢迎大家后续多多交流。\n",
    "\n",
    "**赛题：零基础入门数据挖掘 - 二手车交易价格预测**\n",
    "\n",
    "地址：https://tianchi.aliyun.com/competition/entrance/231784/introduction?spm=5176.12281957.1004.1.38b02448ausjSX "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 模型融合目标\n",
    "\n",
    "* 对于多种调参完成的模型进行模型融合。\n",
    "\n",
    "* 完成对于多种模型的融合，提交融合结果并打卡。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2  内容介绍\n",
    "\n",
    "模型融合是比赛后期一个重要的环节，大体来说有如下的类型方式。\n",
    "\n",
    "1.  简单加权融合:\n",
    "    - 回归（分类概率）：算术平均融合（Arithmetic mean），几何平均融合（Geometric mean）；\n",
    "    - 分类：投票（Voting)\n",
    "    - 综合：排序融合(Rank averaging)，log融合\n",
    "\n",
    "\n",
    "2.  stacking/blending:\n",
    "    - 构建多层模型，并利用预测结果再拟合预测。\n",
    "\n",
    "\n",
    "4.  boosting/bagging（在xgboost，Adaboost,GBDT中已经用到）:\n",
    "    - 多树的提升方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3 Stacking相关理论介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1)  什么是 stacking\n",
    "\n",
    "简单来说 stacking 就是当用初始训练数据学习出若干个基学习器后，将这几个学习器的预测结果作为新的训练集，来学习一个新的学习器。"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://jupter-oss.oss-cn-hangzhou.aliyuncs.com/public/files/image/2326541042/1584448793231_6TygjXwjNb.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将个体学习器结合在一起的时候使用的方法叫做结合策略。对于分类问题，我们可以使用投票法来选择输出最多的类。对于回归问题，我们可以将分类器输出的结果求平均值。\n",
    "\n",
    "上面说的投票法和平均法都是很有效的结合策略，还有一种结合策略是使用另外一个机器学习算法来将个体机器学习器的结果结合在一起，这个方法就是Stacking。\n",
    "\n",
    "在stacking方法中，我们把个体学习器叫做初级学习器，用于结合的学习器叫做次级学习器或元学习器（meta-learner），次级学习器用于训练的数据叫做次级训练集。次级训练集是在训练集上用初级学习器得到的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2)  如何进行 stacking\n",
    "算法示意图如下："
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](http://jupter-oss.oss-cn-hangzhou.aliyuncs.com/public/files/image/2326541042/1584448806789_1ElRtHaacw.jpg)\n",
    "\n",
    "> 引用自 西瓜书《机器学习》"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 过程1-3 是训练出来个体学习器，也就是初级学习器。\n",
    "* 过程5-9是 使用训练出来的个体学习器来得预测的结果，这个预测的结果当做次级学习器的训练集。\n",
    "* 过程11 是用初级学习器预测的结果训练出次级学习器，得到我们最后训练的模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " #### 3）Stacking的方法讲解\n",
    "\n",
    "首先，我们先从一种“不那么正确”但是容易懂的Stacking方法讲起。\n",
    "\n",
    "Stacking模型本质上是一种分层的结构，这里简单起见，只分析二级Stacking.假设我们有2个基模型 Model1_1、Model1_2 和 一个次级模型Model2\n",
    "\n",
    "**Step 1.** 基模型 Model1_1，对训练集train训练，然后用于预测 train 和 test 的标签列，分别是P1，T1\n",
    "\n",
    "Model1_1 模型训练:\n",
    "\n",
    "$$\n",
    "\\left(\\begin{array}{c}{\\vdots} \\\\ {X_{train}} \\\\ {\\vdots}\\end{array}\\right) \\overbrace{\\Longrightarrow}^{\\text {Model1_1 Train} }\\left(\\begin{array}{c}{\\vdots} \\\\ {Y}_{True} \\\\ {\\vdots}\\end{array}\\right)\n",
    "$$\n",
    "\n",
    "训练后的模型 Model1_1 分别在 train 和 test 上预测，得到预测标签分别是P1，T1\n",
    "\n",
    "$$\n",
    "\\left(\\begin{array}{c}{\\vdots} \\\\ {X_{train}} \\\\ {\\vdots}\\end{array}\\right) \\overbrace{\\Longrightarrow}^{\\text {Model1_1 Predict} }\\left(\\begin{array}{c}{\\vdots} \\\\ {P}_{1} \\\\ {\\vdots}\\end{array}\\right)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\left(\\begin{array}{c}{\\vdots} \\\\ {X_{test}} \\\\ {\\vdots}\\end{array}\\right) \\overbrace{\\Longrightarrow}^{\\text {Model1_1 Predict} }\\left(\\begin{array}{c}{\\vdots} \\\\ {T_{1}} \\\\ {\\vdots}\\end{array}\\right)\n",
    "$$\n",
    "\n",
    "**Step 2.** 基模型 Model1_2 ，对训练集train训练，然后用于预测train和test的标签列，分别是P2，T2\n",
    "\n",
    "Model1_2 模型训练:\n",
    "\n",
    "$$\n",
    "\\left(\\begin{array}{c}{\\vdots} \\\\ {X_{train}} \\\\ {\\vdots}\\end{array}\\right) \\overbrace{\\Longrightarrow}^{\\text {Model1_2 Train} }\\left(\\begin{array}{c}{\\vdots} \\\\ {Y}_{True} \\\\ {\\vdots}\\end{array}\\right)\n",
    "$$\n",
    "\n",
    "训练后的模型 Model1_2 分别在 train 和 test 上预测，得到预测标签分别是P2，T2\n",
    "\n",
    "$$\n",
    "\\left(\\begin{array}{c}{\\vdots} \\\\ {X_{train}} \\\\ {\\vdots}\\end{array}\\right) \\overbrace{\\Longrightarrow}^{\\text {Model1_2 Predict} }\\left(\\begin{array}{c}{\\vdots} \\\\ {P}_{2} \\\\ {\\vdots}\\end{array}\\right)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\left(\\begin{array}{c}{\\vdots} \\\\ {X_{test}} \\\\ {\\vdots}\\end{array}\\right) \\overbrace{\\Longrightarrow}^{\\text {Model1_2 Predict} }\\left(\\begin{array}{c}{\\vdots} \\\\ {T_{2}} \\\\ {\\vdots}\\end{array}\\right)\n",
    "$$\n",
    "\n",
    "**Step 3.** 分别把P1,P2以及T1,T2合并，得到一个新的训练集和测试集train2,test2.\n",
    "\n",
    "$$\n",
    "\\overbrace{\\left(\\begin{array}{c}{\\vdots} \\\\ {P_{1}} \\\\ {\\vdots}\\end{array} \\begin{array}{c}{\\vdots} \\\\ {P_{2}} \\\\ {\\vdots}\\end{array} \\right)}^{\\text {Train_2 }}  \n",
    "and \n",
    "\\overbrace{\\left(\\begin{array}{c}{\\vdots} \\\\ {T_{1}} \\\\ {\\vdots}\\end{array} \\begin{array}{c}{\\vdots} \\\\ {T_{2}} \\\\ {\\vdots}\\end{array} \\right)}^{\\text {Test_2 }}\n",
    "$$\n",
    "\n",
    "再用 次级模型 Model2 以真实训练集标签为标签训练,以train2为特征进行训练，预测test2,得到最终的测试集预测的标签列 $Y_{Pre}$。\n",
    "\n",
    "$$\n",
    "\\overbrace{\\left(\\begin{array}{c}{\\vdots} \\\\ {P_{1}} \\\\ {\\vdots}\\end{array} \\begin{array}{c}{\\vdots} \\\\ {P_{2}} \\\\ {\\vdots}\\end{array} \\right)}^{\\text {Train_2 }} \\overbrace{\\Longrightarrow}^{\\text {Model2 Train} }\\left(\\begin{array}{c}{\\vdots} \\\\ {Y}_{True} \\\\ {\\vdots}\\end{array}\\right)\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\overbrace{\\left(\\begin{array}{c}{\\vdots} \\\\ {T_{1}} \\\\ {\\vdots}\\end{array} \\begin{array}{c}{\\vdots} \\\\ {T_{2}} \\\\ {\\vdots}\\end{array} \\right)}^{\\text {Test_2 }} \\overbrace{\\Longrightarrow}^{\\text {Model1_2 Predict} }\\left(\\begin{array}{c}{\\vdots} \\\\ {Y}_{Pre} \\\\ {\\vdots}\\end{array}\\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这就是我们两层堆叠的一种基本的原始思路想法。在不同模型预测的结果基础上再加一层模型，进行再训练，从而得到模型最终的预测。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stacking本质上就是这么直接的思路，但是直接这样有时对于如果训练集和测试集分布不那么一致的情况下是有一点问题的，其问题在于用初始模型训练的标签再利用真实标签进行再训练，毫无疑问会导致一定的模型过拟合训练集，这样或许模型在测试集上的泛化能力或者说效果会有一定的下降，因此现在的问题变成了如何降低再训练的过拟合性，这里我们一般有两种方法。\n",
    "* 1. 次级模型尽量选择简单的线性模型\n",
    "* 2. 利用K折交叉验证"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "K-折交叉验证：\n",
    "训练：\n",
    "\n",
    "![](http://jupter-oss.oss-cn-hangzhou.aliyuncs.com/public/files/image/2326541042/1584448819632_YvJOXMk02P.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "预测：\n",
    "\n",
    "![](http://jupter-oss.oss-cn-hangzhou.aliyuncs.com/public/files/image/2326541042/1584448826203_k8KPy9n7D9.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.4 代码示例"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.1  回归\\分类概率-融合：\n",
    "\n",
    "#### 1）简单加权平均，结果直接融合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 生成一些简单的样本数据，test_prei 代表第i个模型的预测值\n",
    "test_pre1 = [1.2, 3.2, 2.1, 6.2]\n",
    "test_pre2 = [0.9, 3.1, 2.0, 5.9]\n",
    "test_pre3 = [1.1, 2.9, 2.2, 6.0]\n",
    "\n",
    "# y_test_true 代表第模型的真实值\n",
    "y_test_true = [1, 3, 2, 6] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "## 定义结果的加权平均函数\n",
    "def Weighted_method(test_pre1,test_pre2,test_pre3,w=[1/3,1/3,1/3]):\n",
    "    Weighted_result = w[0]*pd.Series(test_pre1)+w[1]*pd.Series(test_pre2)+w[2]*pd.Series(test_pre3)\n",
    "    return Weighted_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pred1 MAE: 0.175\n",
      "Pred2 MAE: 0.075\n",
      "Pred3 MAE: 0.1\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "# 各模型的预测结果计算MAE\n",
    "print('Pred1 MAE:',metrics.mean_absolute_error(y_test_true, test_pre1))\n",
    "print('Pred2 MAE:',metrics.mean_absolute_error(y_test_true, test_pre2))\n",
    "print('Pred3 MAE:',metrics.mean_absolute_error(y_test_true, test_pre3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weighted_pre MAE: 0.0575\n"
     ]
    }
   ],
   "source": [
    "## 根据加权计算MAE\n",
    "w = [0.3,0.4,0.3] # 定义比重权值\n",
    "Weighted_pre = Weighted_method(test_pre1,test_pre2,test_pre3,w)\n",
    "print('Weighted_pre MAE:',metrics.mean_absolute_error(y_test_true, Weighted_pre))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以发现加权结果相对于之前的结果是有提升的，这种我们称其为简单的加权平均。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "还有一些特殊的形式，比如mean平均，median平均"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 定义结果的加权平均函数\n",
    "def Mean_method(test_pre1,test_pre2,test_pre3):\n",
    "    Mean_result = pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Series(test_pre3)],axis=1).mean(axis=1)\n",
    "    return Mean_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean_pre MAE: 0.0666666666667\n"
     ]
    }
   ],
   "source": [
    "Mean_pre = Mean_method(test_pre1,test_pre2,test_pre3)\n",
    "print('Mean_pre MAE:',metrics.mean_absolute_error(y_test_true, Mean_pre))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 定义结果的加权平均函数\n",
    "def Median_method(test_pre1,test_pre2,test_pre3):\n",
    "    Median_result = pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Series(test_pre3)],axis=1).median(axis=1)\n",
    "    return Median_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Median_pre MAE: 0.075\n"
     ]
    }
   ],
   "source": [
    "Median_pre = Median_method(test_pre1,test_pre2,test_pre3)\n",
    "print('Median_pre MAE:',metrics.mean_absolute_error(y_test_true, Median_pre))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  2） Stacking融合(回归)："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model\n",
    "\n",
    "def Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,test_pre1,test_pre2,test_pre3,model_L2= linear_model.LinearRegression()):\n",
    "    model_L2.fit(pd.concat([pd.Series(train_reg1),pd.Series(train_reg2),pd.Series(train_reg3)],axis=1).values,y_train_true)\n",
    "    Stacking_result = model_L2.predict(pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Series(test_pre3)],axis=1).values)\n",
    "    return Stacking_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 生成一些简单的样本数据，test_prei 代表第i个模型的预测值\n",
    "train_reg1 = [3.2, 8.2, 9.1, 5.2]\n",
    "train_reg2 = [2.9, 8.1, 9.0, 4.9]\n",
    "train_reg3 = [3.1, 7.9, 9.2, 5.0]\n",
    "# y_test_true 代表第模型的真实值\n",
    "y_train_true = [3, 8, 9, 5] \n",
    "\n",
    "test_pre1 = [1.2, 3.2, 2.1, 6.2]\n",
    "test_pre2 = [0.9, 3.1, 2.0, 5.9]\n",
    "test_pre3 = [1.1, 2.9, 2.2, 6.0]\n",
    "\n",
    "# y_test_true 代表第模型的真实值\n",
    "y_test_true = [1, 3, 2, 6] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stacking_pre MAE: 0.0421348314607\n"
     ]
    }
   ],
   "source": [
    "model_L2= linear_model.LinearRegression()\n",
    "Stacking_pre = Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,\n",
    "                               test_pre1,test_pre2,test_pre3,model_L2)\n",
    "print('Stacking_pre MAE:',metrics.mean_absolute_error(y_test_true, Stacking_pre))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以发现模型结果相对于之前有进一步的提升，这是我们需要注意的一点是，对于第二层Stacking的模型不宜选取的过于复杂，这样会导致模型在训练集上过拟合，从而使得在测试集上并不能达到很好的效果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.2 分类模型融合：\n",
    "对于分类，同样的可以使用融合方法，比如简单投票，Stacking..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_blobs\n",
    "from sklearn import datasets\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import VotingClassifier\n",
    "from xgboost import XGBClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.datasets import make_moons\n",
    "from sklearn.metrics import accuracy_score,roc_auc_score\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.model_selection import StratifiedKFold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1）Voting投票机制：\n",
    "\n",
    "Voting即投票机制，分为软投票和硬投票两种，其原理采用少数服从多数的思想。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.97 (+/- 0.02) [XGBBoosting]\n",
      "Accuracy: 0.33 (+/- 0.00) [Random Forest]\n",
      "Accuracy: 0.95 (+/- 0.03) [SVM]\n",
      "Accuracy: 0.94 (+/- 0.04) [Ensemble]\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "硬投票：对多个模型直接进行投票，不区分模型结果的相对重要度，最终投票数最多的类为最终被预测的类。\n",
    "'''\n",
    "iris = datasets.load_iris()\n",
    "\n",
    "x=iris.data\n",
    "y=iris.target\n",
    "x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.3)\n",
    "\n",
    "clf1 = XGBClassifier(learning_rate=0.1, n_estimators=150, max_depth=3, min_child_weight=2, subsample=0.7,\n",
    "                     colsample_bytree=0.6, objective='binary:logistic')\n",
    "clf2 = RandomForestClassifier(n_estimators=50, max_depth=1, min_samples_split=4,\n",
    "                              min_samples_leaf=63,oob_score=True)\n",
    "clf3 = SVC(C=0.1)\n",
    "\n",
    "# 硬投票\n",
    "eclf = VotingClassifier(estimators=[('xgb', clf1), ('rf', clf2), ('svc', clf3)], voting='hard')\n",
    "for clf, label in zip([clf1, clf2, clf3, eclf], ['XGBBoosting', 'Random Forest', 'SVM', 'Ensemble']):\n",
    "    scores = cross_val_score(clf, x, y, cv=5, scoring='accuracy')\n",
    "    print(\"Accuracy: %0.2f (+/- %0.2f) [%s]\" % (scores.mean(), scores.std(), label))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.96 (+/- 0.02) [XGBBoosting]\n",
      "Accuracy: 0.33 (+/- 0.00) [Random Forest]\n",
      "Accuracy: 0.95 (+/- 0.03) [SVM]\n",
      "Accuracy: 0.96 (+/- 0.02) [Ensemble]\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "软投票：和硬投票原理相同，增加了设置权重的功能，可以为不同模型设置不同权重，进而区别模型不同的重要度。\n",
    "'''\n",
    "x=iris.data\n",
    "y=iris.target\n",
    "x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.3)\n",
    "\n",
    "clf1 = XGBClassifier(learning_rate=0.1, n_estimators=150, max_depth=3, min_child_weight=2, subsample=0.8,\n",
    "                     colsample_bytree=0.8, objective='binary:logistic')\n",
    "clf2 = RandomForestClassifier(n_estimators=50, max_depth=1, min_samples_split=4,\n",
    "                              min_samples_leaf=63,oob_score=True)\n",
    "clf3 = SVC(C=0.1, probability=True)\n",
    "\n",
    "# 软投票\n",
    "eclf = VotingClassifier(estimators=[('xgb', clf1), ('rf', clf2), ('svc', clf3)], voting='soft', weights=[2, 1, 1])\n",
    "clf1.fit(x_train, y_train)\n",
    "\n",
    "for clf, label in zip([clf1, clf2, clf3, eclf], ['XGBBoosting', 'Random Forest', 'SVM', 'Ensemble']):\n",
    "    scores = cross_val_score(clf, x, y, cv=5, scoring='accuracy')\n",
    "    print(\"Accuracy: %0.2f (+/- %0.2f) [%s]\" % (scores.mean(), scores.std(), label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2）分类的Stacking\\Blending融合："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "stacking是一种分层模型集成框架。\n",
    "\n",
    "> 以两层为例，第一层由多个基学习器组成，其输入为原始训练集，第二层的模型则是以第一层基学习器的输出作为训练集进行再训练，从而得到完整的stacking模型, stacking两层模型都使用了全部的训练数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "val auc Score: 1.000000\n",
      "val auc Score: 0.500000\n",
      "val auc Score: 0.500000\n",
      "val auc Score: 0.500000\n",
      "val auc Score: 0.500000\n",
      "Val auc Score of Stacking: 1.000000\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "5-Fold Stacking\n",
    "'''\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import ExtraTreesClassifier,GradientBoostingClassifier\n",
    "import pandas as pd\n",
    "#创建训练的数据集\n",
    "data_0 = iris.data\n",
    "data = data_0[:100,:]\n",
    "\n",
    "target_0 = iris.target\n",
    "target = target_0[:100]\n",
    "\n",
    "#模型融合中使用到的各个单模型\n",
    "clfs = [LogisticRegression(solver='lbfgs'),\n",
    "        RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),\n",
    "        ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),\n",
    "        ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),\n",
    "        GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)]\n",
    " \n",
    "#切分一部分数据作为测试集\n",
    "X, X_predict, y, y_predict = train_test_split(data, target, test_size=0.3, random_state=2020)\n",
    "\n",
    "dataset_blend_train = np.zeros((X.shape[0], len(clfs)))\n",
    "dataset_blend_test = np.zeros((X_predict.shape[0], len(clfs)))\n",
    "\n",
    "#5折stacking\n",
    "n_splits = 5\n",
    "skf = StratifiedKFold(n_splits)\n",
    "skf = skf.split(X, y)\n",
    "\n",
    "for j, clf in enumerate(clfs):\n",
    "    #依次训练各个单模型\n",
    "    dataset_blend_test_j = np.zeros((X_predict.shape[0], 5))\n",
    "    for i, (train, test) in enumerate(skf):\n",
    "        #5-Fold交叉训练，使用第i个部分作为预测，剩余的部分来训练模型，获得其预测的输出作为第i部分的新特征。\n",
    "        X_train, y_train, X_test, y_test = X[train], y[train], X[test], y[test]\n",
    "        clf.fit(X_train, y_train)\n",
    "        y_submission = clf.predict_proba(X_test)[:, 1]\n",
    "        dataset_blend_train[test, j] = y_submission\n",
    "        dataset_blend_test_j[:, i] = clf.predict_proba(X_predict)[:, 1]\n",
    "    #对于测试集，直接用这k个模型的预测值均值作为新的特征。\n",
    "    dataset_blend_test[:, j] = dataset_blend_test_j.mean(1)\n",
    "    print(\"val auc Score: %f\" % roc_auc_score(y_predict, dataset_blend_test[:, j]))\n",
    "\n",
    "clf = LogisticRegression(solver='lbfgs')\n",
    "clf.fit(dataset_blend_train, y)\n",
    "y_submission = clf.predict_proba(dataset_blend_test)[:, 1]\n",
    "\n",
    "print(\"Val auc Score of Stacking: %f\" % (roc_auc_score(y_predict, y_submission)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Blending，其实和Stacking是一种类似的多层模型融合的形式\n",
    "\n",
    "> 其主要思路是把原始的训练集先分成两部分，比如70%的数据作为新的训练集，剩下30%的数据作为测试集。\n",
    "\n",
    "> 在第一层，我们在这70%的数据上训练多个模型，然后去预测那30%数据的label，同时也预测test集的label。\n",
    "\n",
    "> 在第二层，我们就直接用这30%数据在第一层预测的结果做为新特征继续训练，然后用test集第一层预测的label做特征，用第二层训练的模型做进一步预测\n",
    "\n",
    "其优点在于：\n",
    "* 1.比stacking简单（因为不用进行k次的交叉验证来获得stacker feature）\n",
    "* 2.避开了一个信息泄露问题：generlizers和stacker使用了不一样的数据集\n",
    "\n",
    "缺点在于：\n",
    "* 1.使用了很少的数据（第二阶段的blender只使用training set10%的量）\n",
    "* 2.blender可能会过拟合\n",
    "* 3.stacking使用多次的交叉验证会比较稳健\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "val auc Score: 1.000000\n",
      "val auc Score: 1.000000\n",
      "val auc Score: 1.000000\n",
      "val auc Score: 1.000000\n",
      "val auc Score: 1.000000\n",
      "Val auc Score of Blending: 1.000000\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "Blending\n",
    "'''\n",
    " \n",
    "#创建训练的数据集\n",
    "#创建训练的数据集\n",
    "data_0 = iris.data\n",
    "data = data_0[:100,:]\n",
    "\n",
    "target_0 = iris.target\n",
    "target = target_0[:100]\n",
    " \n",
    "#模型融合中使用到的各个单模型\n",
    "clfs = [LogisticRegression(solver='lbfgs'),\n",
    "        RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),\n",
    "        RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),\n",
    "        ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),\n",
    "        #ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),\n",
    "        GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)]\n",
    " \n",
    "#切分一部分数据作为测试集\n",
    "X, X_predict, y, y_predict = train_test_split(data, target, test_size=0.3, random_state=2020)\n",
    "\n",
    "#切分训练数据集为d1,d2两部分\n",
    "X_d1, X_d2, y_d1, y_d2 = train_test_split(X, y, test_size=0.5, random_state=2020)\n",
    "dataset_d1 = np.zeros((X_d2.shape[0], len(clfs)))\n",
    "dataset_d2 = np.zeros((X_predict.shape[0], len(clfs)))\n",
    " \n",
    "for j, clf in enumerate(clfs):\n",
    "    #依次训练各个单模型\n",
    "    clf.fit(X_d1, y_d1)\n",
    "    y_submission = clf.predict_proba(X_d2)[:, 1]\n",
    "    dataset_d1[:, j] = y_submission\n",
    "    #对于测试集，直接用这k个模型的预测值作为新的特征。\n",
    "    dataset_d2[:, j] = clf.predict_proba(X_predict)[:, 1]\n",
    "    print(\"val auc Score: %f\" % roc_auc_score(y_predict, dataset_d2[:, j]))\n",
    "\n",
    "#融合使用的模型\n",
    "clf = GradientBoostingClassifier(learning_rate=0.02, subsample=0.5, max_depth=6, n_estimators=30)\n",
    "clf.fit(dataset_d1, y_d2)\n",
    "y_submission = clf.predict_proba(dataset_d2)[:, 1]\n",
    "print(\"Val auc Score of Blending: %f\" % (roc_auc_score(y_predict, y_submission)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参考博客：https://blog.csdn.net/Noob_daniel/article/details/76087829"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3）分类的Stacking融合(利用mlxtend)："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install mlxtend\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "import itertools\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.gridspec as gridspec\n",
    "\n",
    "from sklearn import datasets\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.naive_bayes import GaussianNB \n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from mlxtend.classifier import StackingClassifier\n",
    "\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from mlxtend.plotting import plot_learning_curves\n",
    "from mlxtend.plotting import plot_decision_regions\n",
    "\n",
    "# 以python自带的鸢尾花数据集为例\n",
    "iris = datasets.load_iris()\n",
    "X, y = iris.data[:, 1:3], iris.target\n",
    "\n",
    "clf1 = KNeighborsClassifier(n_neighbors=1)\n",
    "clf2 = RandomForestClassifier(random_state=1)\n",
    "clf3 = GaussianNB()\n",
    "lr = LogisticRegression()\n",
    "sclf = StackingClassifier(classifiers=[clf1, clf2, clf3], \n",
    "                          meta_classifier=lr)\n",
    "\n",
    "label = ['KNN', 'Random Forest', 'Naive Bayes', 'Stacking Classifier']\n",
    "clf_list = [clf1, clf2, clf3, sclf]\n",
    "\n",
    "fig = plt.figure(figsize=(10,8))\n",
    "gs = gridspec.GridSpec(2, 2)\n",
    "grid = itertools.product([0,1],repeat=2)\n",
    "\n",
    "clf_cv_mean = []\n",
    "clf_cv_std = []\n",
    "for clf, label, grd in zip(clf_list, label, grid):\n",
    "        \n",
    "    scores = cross_val_score(clf, X, y, cv=3, scoring='accuracy')\n",
    "    print(\"Accuracy: %.2f (+/- %.2f) [%s]\" %(scores.mean(), scores.std(), label))\n",
    "    clf_cv_mean.append(scores.mean())\n",
    "    clf_cv_std.append(scores.std())\n",
    "        \n",
    "    clf.fit(X, y)\n",
    "    ax = plt.subplot(gs[grd[0], grd[1]])\n",
    "    fig = plot_decision_regions(X=X, y=y, clf=clf)\n",
    "    plt.title(label)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以发现 基模型 用 'KNN', 'Random Forest', 'Naive Bayes' 然后再这基础上 次级模型加一个 'LogisticRegression'，模型测试效果有着很好的提升。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.3 一些其它方法："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 将特征放进模型中预测，并将预测结果变换并作为新的特征加入原有特征中再经过模型预测结果 （Stacking变化）\n",
    "\n",
    "（可以反复预测多次将结果加入最后的特征中）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-12-24T12:50:55.020619Z",
     "start_time": "2019-12-24T12:50:55.004659Z"
    }
   },
   "outputs": [],
   "source": [
    "def Ensemble_add_feature(train,test,target,clfs):\n",
    "    \n",
    "    # n_flods = 5\n",
    "    # skf = list(StratifiedKFold(y, n_folds=n_flods))\n",
    "\n",
    "    train_ = np.zeros((train.shape[0],len(clfs*2)))\n",
    "    test_ = np.zeros((test.shape[0],len(clfs*2)))\n",
    "\n",
    "    for j,clf in enumerate(clfs):\n",
    "        '''依次训练各个单模型'''\n",
    "        # print(j, clf)\n",
    "        '''使用第1个部分作为预测，第2部分来训练模型，获得其预测的输出作为第2部分的新特征。'''\n",
    "        # X_train, y_train, X_test, y_test = X[train], y[train], X[test], y[test]\n",
    "\n",
    "        clf.fit(train,target)\n",
    "        y_train = clf.predict(train)\n",
    "        y_test = clf.predict(test)\n",
    "\n",
    "        ## 新特征生成\n",
    "        train_[:,j*2] = y_train**2\n",
    "        test_[:,j*2] = y_test**2\n",
    "        train_[:, j+1] = np.exp(y_train)\n",
    "        test_[:, j+1] = np.exp(y_test)\n",
    "        # print(\"val auc Score: %f\" % r2_score(y_predict, dataset_d2[:, j]))\n",
    "        print('Method ',j)\n",
    "    \n",
    "    train_ = pd.DataFrame(train_)\n",
    "    test_ = pd.DataFrame(test_)\n",
    "    return train_,test_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Method  0\n",
      "Method  1\n",
      "Method  2\n",
      "Method  3\n",
      "Method  4\n",
      "Val auc Score of stacking: 1.000000\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score, train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "clf = LogisticRegression()\n",
    "\n",
    "data_0 = iris.data\n",
    "data = data_0[:100,:]\n",
    "\n",
    "target_0 = iris.target\n",
    "target = target_0[:100]\n",
    "\n",
    "x_train,x_test,y_train,y_test=train_test_split(data,target,test_size=0.3)\n",
    "x_train = pd.DataFrame(x_train) ; x_test = pd.DataFrame(x_test)\n",
    "\n",
    "#模型融合中使用到的各个单模型\n",
    "clfs = [LogisticRegression(),\n",
    "        RandomForestClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),\n",
    "        ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='gini'),\n",
    "        ExtraTreesClassifier(n_estimators=5, n_jobs=-1, criterion='entropy'),\n",
    "        GradientBoostingClassifier(learning_rate=0.05, subsample=0.5, max_depth=6, n_estimators=5)]\n",
    "\n",
    "New_train,New_test = Ensemble_add_feature(x_train,x_test,y_train,clfs)\n",
    "\n",
    "clf = LogisticRegression()\n",
    "# clf = GradientBoostingClassifier(learning_rate=0.02, subsample=0.5, max_depth=6, n_estimators=30)\n",
    "clf.fit(New_train, y_train)\n",
    "y_emb = clf.predict_proba(New_test)[:, 1]\n",
    "\n",
    "print(\"Val auc Score of stacking: %f\" % (roc_auc_score(y_test, y_emb)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.4.4 本赛题示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import warnings\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "warnings.filterwarnings('ignore')\n",
    "%matplotlib inline\n",
    "\n",
    "import itertools\n",
    "import matplotlib.gridspec as gridspec\n",
    "from sklearn import datasets\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.naive_bayes import GaussianNB \n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "# from mlxtend.classifier import StackingClassifier\n",
    "from sklearn.model_selection import cross_val_score, train_test_split\n",
    "# from mlxtend.plotting import plot_learning_curves\n",
    "# from mlxtend.plotting import plot_decision_regions\n",
    "\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from sklearn import linear_model\n",
    "from sklearn import preprocessing\n",
    "from sklearn.svm import SVR\n",
    "from sklearn.decomposition import PCA,FastICA,FactorAnalysis,SparsePCA\n",
    "\n",
    "import lightgbm as lgb\n",
    "import xgboost as xgb\n",
    "from sklearn.model_selection import GridSearchCV,cross_val_score\n",
    "from sklearn.ensemble import RandomForestRegressor,GradientBoostingRegressor\n",
    "\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(150000, 31)\n",
      "(50000, 30)\n"
     ]
    }
   ],
   "source": [
    "## 数据读取\n",
    "Train_data = pd.read_csv('datalab/231784/used_car_train_20200313.csv', sep=' ')\n",
    "TestA_data = pd.read_csv('datalab/231784/used_car_testA_20200313.csv', sep=' ')\n",
    "\n",
    "print(Train_data.shape)\n",
    "print(TestA_data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SaleID</th>\n",
       "      <th>name</th>\n",
       "      <th>regDate</th>\n",
       "      <th>model</th>\n",
       "      <th>brand</th>\n",
       "      <th>bodyType</th>\n",
       "      <th>fuelType</th>\n",
       "      <th>gearbox</th>\n",
       "      <th>power</th>\n",
       "      <th>kilometer</th>\n",
       "      <th>...</th>\n",
       "      <th>v_5</th>\n",
       "      <th>v_6</th>\n",
       "      <th>v_7</th>\n",
       "      <th>v_8</th>\n",
       "      <th>v_9</th>\n",
       "      <th>v_10</th>\n",
       "      <th>v_11</th>\n",
       "      <th>v_12</th>\n",
       "      <th>v_13</th>\n",
       "      <th>v_14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>736</td>\n",
       "      <td>20040402</td>\n",
       "      <td>30.0</td>\n",
       "      <td>6</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>60</td>\n",
       "      <td>12.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.235676</td>\n",
       "      <td>0.101988</td>\n",
       "      <td>0.129549</td>\n",
       "      <td>0.022816</td>\n",
       "      <td>0.097462</td>\n",
       "      <td>-2.881803</td>\n",
       "      <td>2.804097</td>\n",
       "      <td>-2.420821</td>\n",
       "      <td>0.795292</td>\n",
       "      <td>0.914762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2262</td>\n",
       "      <td>20030301</td>\n",
       "      <td>40.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.264777</td>\n",
       "      <td>0.121004</td>\n",
       "      <td>0.135731</td>\n",
       "      <td>0.026597</td>\n",
       "      <td>0.020582</td>\n",
       "      <td>-4.900482</td>\n",
       "      <td>2.096338</td>\n",
       "      <td>-1.030483</td>\n",
       "      <td>-1.722674</td>\n",
       "      <td>0.245522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>14874</td>\n",
       "      <td>20040403</td>\n",
       "      <td>115.0</td>\n",
       "      <td>15</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>163</td>\n",
       "      <td>12.5</td>\n",
       "      <td>...</td>\n",
       "      <td>0.251410</td>\n",
       "      <td>0.114912</td>\n",
       "      <td>0.165147</td>\n",
       "      <td>0.062173</td>\n",
       "      <td>0.027075</td>\n",
       "      <td>-4.846749</td>\n",
       "      <td>1.803559</td>\n",
       "      <td>1.565330</td>\n",
       "      <td>-0.832687</td>\n",
       "      <td>-0.229963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>71865</td>\n",
       "      <td>19960908</td>\n",
       "      <td>109.0</td>\n",
       "      <td>10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>193</td>\n",
       "      <td>15.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.274293</td>\n",
       "      <td>0.110300</td>\n",
       "      <td>0.121964</td>\n",
       "      <td>0.033395</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-4.509599</td>\n",
       "      <td>1.285940</td>\n",
       "      <td>-0.501868</td>\n",
       "      <td>-2.438353</td>\n",
       "      <td>-0.478699</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>111080</td>\n",
       "      <td>20120103</td>\n",
       "      <td>110.0</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>68</td>\n",
       "      <td>5.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.228036</td>\n",
       "      <td>0.073205</td>\n",
       "      <td>0.091880</td>\n",
       "      <td>0.078819</td>\n",
       "      <td>0.121534</td>\n",
       "      <td>-1.896240</td>\n",
       "      <td>0.910783</td>\n",
       "      <td>0.931110</td>\n",
       "      <td>2.834518</td>\n",
       "      <td>1.923482</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   SaleID    name   regDate  model  brand  bodyType  fuelType  gearbox  power  \\\n",
       "0       0     736  20040402   30.0      6       1.0       0.0      0.0     60   \n",
       "1       1    2262  20030301   40.0      1       2.0       0.0      0.0      0   \n",
       "2       2   14874  20040403  115.0     15       1.0       0.0      0.0    163   \n",
       "3       3   71865  19960908  109.0     10       0.0       0.0      1.0    193   \n",
       "4       4  111080  20120103  110.0      5       1.0       0.0      0.0     68   \n",
       "\n",
       "   kilometer    ...          v_5       v_6       v_7       v_8       v_9  \\\n",
       "0       12.5    ...     0.235676  0.101988  0.129549  0.022816  0.097462   \n",
       "1       15.0    ...     0.264777  0.121004  0.135731  0.026597  0.020582   \n",
       "2       12.5    ...     0.251410  0.114912  0.165147  0.062173  0.027075   \n",
       "3       15.0    ...     0.274293  0.110300  0.121964  0.033395  0.000000   \n",
       "4        5.0    ...     0.228036  0.073205  0.091880  0.078819  0.121534   \n",
       "\n",
       "       v_10      v_11      v_12      v_13      v_14  \n",
       "0 -2.881803  2.804097 -2.420821  0.795292  0.914762  \n",
       "1 -4.900482  2.096338 -1.030483 -1.722674  0.245522  \n",
       "2 -4.846749  1.803559  1.565330 -0.832687 -0.229963  \n",
       "3 -4.509599  1.285940 -0.501868 -2.438353 -0.478699  \n",
       "4 -1.896240  0.910783  0.931110  2.834518  1.923482  \n",
       "\n",
       "[5 rows x 31 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['SaleID', 'name', 'regDate', 'model', 'brand', 'bodyType', 'fuelType',\n",
      "       'gearbox', 'power', 'kilometer', 'regionCode', 'seller', 'offerType',\n",
      "       'creatDate', 'price', 'v_0', 'v_1', 'v_2', 'v_3', 'v_4', 'v_5', 'v_6',\n",
      "       'v_7', 'v_8', 'v_9', 'v_10', 'v_11', 'v_12', 'v_13', 'v_14'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "numerical_cols = Train_data.select_dtypes(exclude = 'object').columns\n",
    "print(numerical_cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_cols = [col for col in numerical_cols if col not in ['SaleID','name','regDate','price']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X train shape: (150000, 26)\n",
      "X test shape: (50000, 26)\n"
     ]
    }
   ],
   "source": [
    "X_data = Train_data[feature_cols]\n",
    "Y_data = Train_data['price']\n",
    "\n",
    "X_test  = TestA_data[feature_cols]\n",
    "\n",
    "print('X train shape:',X_data.shape)\n",
    "print('X test shape:',X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Sta_inf(data):\n",
    "    print('_min',np.min(data))\n",
    "    print('_max:',np.max(data))\n",
    "    print('_mean',np.mean(data))\n",
    "    print('_ptp',np.ptp(data))\n",
    "    print('_std',np.std(data))\n",
    "    print('_var',np.var(data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sta of label:\n",
      "_min 11\n",
      "_max: 99999\n",
      "_mean 5923.32733333\n",
      "_ptp 99988\n",
      "_std 7501.97346988\n",
      "_var 56279605.9427\n"
     ]
    }
   ],
   "source": [
    "print('Sta of label:')\n",
    "Sta_inf(Y_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_data = X_data.fillna(-1)\n",
    "X_test = X_test.fillna(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model_lr(x_train,y_train):\n",
    "    reg_model = linear_model.LinearRegression()\n",
    "    reg_model.fit(x_train,y_train)\n",
    "    return reg_model\n",
    "\n",
    "def build_model_ridge(x_train,y_train):\n",
    "    reg_model = linear_model.Ridge(alpha=0.8)#alphas=range(1,100,5)\n",
    "    reg_model.fit(x_train,y_train)\n",
    "    return reg_model\n",
    "\n",
    "def build_model_lasso(x_train,y_train):\n",
    "    reg_model = linear_model.LassoCV()\n",
    "    reg_model.fit(x_train,y_train)\n",
    "    return reg_model\n",
    "\n",
    "def build_model_gbdt(x_train,y_train):\n",
    "    estimator =GradientBoostingRegressor(loss='ls',subsample= 0.85,max_depth= 5,n_estimators = 100)\n",
    "    param_grid = { \n",
    "            'learning_rate': [0.05,0.08,0.1,0.2],\n",
    "            }\n",
    "    gbdt = GridSearchCV(estimator, param_grid,cv=3)\n",
    "    gbdt.fit(x_train,y_train)\n",
    "    print(gbdt.best_params_)\n",
    "    # print(gbdt.best_estimator_ )\n",
    "    return gbdt\n",
    "\n",
    "def build_model_xgb(x_train,y_train):\n",
    "    model = xgb.XGBRegressor(n_estimators=120, learning_rate=0.08, gamma=0, subsample=0.8,\\\n",
    "        colsample_bytree=0.9, max_depth=5) #, objective ='reg:squarederror'\n",
    "    model.fit(x_train, y_train)\n",
    "    return model\n",
    "\n",
    "def build_model_lgb(x_train,y_train):\n",
    "    estimator = lgb.LGBMRegressor(num_leaves=63,n_estimators = 100)\n",
    "    param_grid = {\n",
    "        'learning_rate': [0.01, 0.05, 0.1],\n",
    "    }\n",
    "    gbm = GridSearchCV(estimator, param_grid)\n",
    "    gbm.fit(x_train, y_train)\n",
    "    return gbm\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2）XGBoost的五折交叉回归验证实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train mae: 558.212360169\n",
      "Val mae 693.120168439\n"
     ]
    }
   ],
   "source": [
    "## xgb\n",
    "xgr = xgb.XGBRegressor(n_estimators=120, learning_rate=0.1, subsample=0.8,\\\n",
    "        colsample_bytree=0.9, max_depth=7) # ,objective ='reg:squarederror'\n",
    "\n",
    "scores_train = []\n",
    "scores = []\n",
    "\n",
    "## 5折交叉验证方式\n",
    "sk=StratifiedKFold(n_splits=5,shuffle=True,random_state=0)\n",
    "for train_ind,val_ind in sk.split(X_data,Y_data):\n",
    "    \n",
    "    train_x=X_data.iloc[train_ind].values\n",
    "    train_y=Y_data.iloc[train_ind]\n",
    "    val_x=X_data.iloc[val_ind].values\n",
    "    val_y=Y_data.iloc[val_ind]\n",
    "    \n",
    "    xgr.fit(train_x,train_y)\n",
    "    pred_train_xgb=xgr.predict(train_x)\n",
    "    pred_xgb=xgr.predict(val_x)\n",
    "    \n",
    "    score_train = mean_absolute_error(train_y,pred_train_xgb)\n",
    "    scores_train.append(score_train)\n",
    "    score = mean_absolute_error(val_y,pred_xgb)\n",
    "    scores.append(score)\n",
    "\n",
    "print('Train mae:',np.mean(score_train))\n",
    "print('Val mae',np.mean(scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3）划分数据集，并用多种方法训练和预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predict LR...\n",
      "Predict Ridge...\n",
      "Predict Lasso...\n",
      "Predict GBDT...\n",
      "{'learning_rate': 0.1, 'n_estimators': 80}\n"
     ]
    }
   ],
   "source": [
    "## Split data with val\n",
    "x_train,x_val,y_train,y_val = train_test_split(X_data,Y_data,test_size=0.3)\n",
    "\n",
    "## Train and Predict\n",
    "print('Predict LR...')\n",
    "model_lr = build_model_lr(x_train,y_train)\n",
    "val_lr = model_lr.predict(x_val)\n",
    "subA_lr = model_lr.predict(X_test)\n",
    "\n",
    "print('Predict Ridge...')\n",
    "model_ridge = build_model_ridge(x_train,y_train)\n",
    "val_ridge = model_ridge.predict(x_val)\n",
    "subA_ridge = model_ridge.predict(X_test)\n",
    "\n",
    "print('Predict Lasso...')\n",
    "model_lasso = build_model_lasso(x_train,y_train)\n",
    "val_lasso = model_lasso.predict(x_val)\n",
    "subA_lasso = model_lasso.predict(X_test)\n",
    "\n",
    "print('Predict GBDT...')\n",
    "model_gbdt = build_model_gbdt(x_train,y_train)\n",
    "val_gbdt = model_gbdt.predict(x_val)\n",
    "subA_gbdt = model_gbdt.predict(X_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 一般比赛中效果最为显著的两种方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predict XGB...\n",
      "predict lgb...\n"
     ]
    }
   ],
   "source": [
    "print('predict XGB...')\n",
    "model_xgb = build_model_xgb(x_train,y_train)\n",
    "val_xgb = model_xgb.predict(x_val)\n",
    "subA_xgb = model_xgb.predict(X_test)\n",
    "\n",
    "print('predict lgb...')\n",
    "model_lgb = build_model_lgb(x_train,y_train)\n",
    "val_lgb = model_lgb.predict(x_val)\n",
    "subA_lgb = model_lgb.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sta inf of lgb:\n",
      "_min -126.864734992\n",
      "_max: 90152.4775557\n",
      "_mean 5917.96632163\n",
      "_ptp 90279.3422907\n",
      "_std 7358.88582391\n",
      "_var 54153200.5693\n"
     ]
    }
   ],
   "source": [
    "print('Sta inf of lgb:')\n",
    "Sta_inf(subA_lgb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  1）加权融合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE of Weighted of val: 730.877443666\n",
      "Sta inf:\n",
      "_min -2816.93914153\n",
      "_max: 88576.7842223\n",
      "_mean 5920.38233546\n",
      "_ptp 91393.7233639\n",
      "_std 7325.20946801\n",
      "_var 53658693.7502\n"
     ]
    }
   ],
   "source": [
    "def Weighted_method(test_pre1,test_pre2,test_pre3,w=[1/3,1/3,1/3]):\n",
    "    Weighted_result = w[0]*pd.Series(test_pre1)+w[1]*pd.Series(test_pre2)+w[2]*pd.Series(test_pre3)\n",
    "    return Weighted_result\n",
    "\n",
    "## Init the Weight\n",
    "w = [0.3,0.4,0.3]\n",
    "\n",
    "## 测试验证集准确度\n",
    "val_pre = Weighted_method(val_lgb,val_xgb,val_gbdt,w)\n",
    "MAE_Weighted = mean_absolute_error(y_val,val_pre)\n",
    "print('MAE of Weighted of val:',MAE_Weighted)\n",
    "\n",
    "## 预测数据部分\n",
    "subA = Weighted_method(subA_lgb,subA_xgb,subA_gbdt,w)\n",
    "print('Sta inf:')\n",
    "Sta_inf(subA)\n",
    "## 生成提交文件\n",
    "sub = pd.DataFrame()\n",
    "sub['SaleID'] = X_test.index\n",
    "sub['price'] = subA\n",
    "sub.to_csv('./sub_Weighted.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE of lr: 2597.45638384\n"
     ]
    }
   ],
   "source": [
    "## 与简单的LR（线性回归）进行对比\n",
    "val_lr_pred = model_lr.predict(x_val)\n",
    "MAE_lr = mean_absolute_error(y_val,val_lr_pred)\n",
    "print('MAE of lr:',MAE_lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  2）Starking融合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Starking\n",
    "\n",
    "## 第一层\n",
    "train_lgb_pred = model_lgb.predict(x_train)\n",
    "train_xgb_pred = model_xgb.predict(x_train)\n",
    "train_gbdt_pred = model_gbdt.predict(x_train)\n",
    "\n",
    "Strak_X_train = pd.DataFrame()\n",
    "Strak_X_train['Method_1'] = train_lgb_pred\n",
    "Strak_X_train['Method_2'] = train_xgb_pred\n",
    "Strak_X_train['Method_3'] = train_gbdt_pred\n",
    "\n",
    "Strak_X_val = pd.DataFrame()\n",
    "Strak_X_val['Method_1'] = val_lgb\n",
    "Strak_X_val['Method_2'] = val_xgb\n",
    "Strak_X_val['Method_3'] = val_gbdt\n",
    "\n",
    "Strak_X_test = pd.DataFrame()\n",
    "Strak_X_test['Method_1'] = subA_lgb\n",
    "Strak_X_test['Method_2'] = subA_xgb\n",
    "Strak_X_test['Method_3'] = subA_gbdt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Method_1</th>\n",
       "      <th>Method_2</th>\n",
       "      <th>Method_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39682.037093</td>\n",
       "      <td>41029.078125</td>\n",
       "      <td>40552.596813</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>239.498371</td>\n",
       "      <td>266.032654</td>\n",
       "      <td>393.909761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6915.162439</td>\n",
       "      <td>7345.680664</td>\n",
       "      <td>7623.552178</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11861.783785</td>\n",
       "      <td>11721.493164</td>\n",
       "      <td>11463.293245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>583.773267</td>\n",
       "      <td>513.307983</td>\n",
       "      <td>520.665295</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Method_1      Method_2      Method_3\n",
       "0  39682.037093  41029.078125  40552.596813\n",
       "1    239.498371    266.032654    393.909761\n",
       "2   6915.162439   7345.680664   7623.552178\n",
       "3  11861.783785  11721.493164  11463.293245\n",
       "4    583.773267    513.307983    520.665295"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Strak_X_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MAE of Stacking-LR: 628.399441036\n",
      "MAE of Stacking-LR: 707.673951794\n",
      "Predict Stacking-LR...\n"
     ]
    }
   ],
   "source": [
    "## level2-method \n",
    "model_lr_Stacking = build_model_lr(Strak_X_train,y_train)\n",
    "## 训练集\n",
    "train_pre_Stacking = model_lr_Stacking.predict(Strak_X_train)\n",
    "print('MAE of Stacking-LR:',mean_absolute_error(y_train,train_pre_Stacking))\n",
    "\n",
    "## 验证集\n",
    "val_pre_Stacking = model_lr_Stacking.predict(Strak_X_val)\n",
    "print('MAE of Stacking-LR:',mean_absolute_error(y_val,val_pre_Stacking))\n",
    "\n",
    "## 预测集\n",
    "print('Predict Stacking-LR...')\n",
    "subA_Stacking = model_lr_Stacking.predict(Strak_X_test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "subA_Stacking[subA_Stacking<10]=10  ## 去除过小的预测值\n",
    "\n",
    "sub = pd.DataFrame()\n",
    "sub['SaleID'] = X_test.index\n",
    "sub['price'] = subA_Stacking\n",
    "sub.to_csv('./sub_Stacking.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sta inf:\n",
      "_min 10.0\n",
      "_max: 90849.3729816\n",
      "_mean 5917.39429976\n",
      "_ptp 90839.3729816\n",
      "_std 7396.09766172\n",
      "_var 54702260.6217\n"
     ]
    }
   ],
   "source": [
    "print('Sta inf:')\n",
    "Sta_inf(subA_Stacking)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 经验总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "比赛的融合这个问题，个人的看法来说其实涉及多个层面，也是提分和提升模型鲁棒性的一种重要方法：\n",
    "\n",
    "* 1）**结果层面的融合**，这种是最常见的融合方法，其可行的融合方法也有很多，比如根据结果的得分进行加权融合，还可以做Log，exp处理等。在做结果融合的时候，有一个很重要的条件是模型结果的得分要比较近似，然后结果的差异要比较大，这样的结果融合往往有比较好的效果提升。\n",
    "\n",
    "* 2）**特征层面的融合**，这个层面其实感觉不叫融合，准确说可以叫分割，很多时候如果我们用同种模型训练，可以把特征进行切分给不同的模型，然后在后面进行模型或者结果融合有时也能产生比较好的效果。\n",
    "\n",
    "* 3）**模型层面的融合**，模型层面的融合可能就涉及模型的堆叠和设计，比如加Staking层，部分模型的结果作为特征输入等，这些就需要多实验和思考了，基于模型层面的融合最好不同模型类型要有一定的差异，用同种模型不同的参数的收益一般是比较小的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task 5-模型融合 END.**\n",
    "\n",
    "--- By: ML67 \n",
    "\n",
    "        Email: maolinw67@163.com\n",
    "        PS: 华中科技大学研究生, 长期混迹Tianchi等，希望和大家多多交流。\n",
    "        github: https://github.com/mlw67 （近期会做一些书籍推导和代码的整理）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**关于Datawhale：**\n",
    "\n",
    "> Datawhale是一个专注于数据科学与AI领域的开源组织，汇集了众多领域院校和知名企业的优秀学习者，聚合了一群有开源精神和探索精神的团队成员。Datawhale 以“for the learner，和学习者一起成长”为愿景，鼓励真实地展现自我、开放包容、互信互助、敢于试错和勇于担当。同时 Datawhale 用开源的理念去探索开源内容、开源学习和开源方案，赋能人才培养，助力人才成长，建立起人与人，人与知识，人与企业和人与未来的联结。\n",
    "\n",
    "本次数据挖掘路径学习，专题知识将在天池分享，详情可关注Datawhale：\n",
    "\n",
    "![](http://jupter-oss.oss-cn-hangzhou.aliyuncs.com/public/files/image/2326541042/1584426326920_9FOUExG2be.jpg)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "307.2px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
